A unified algorithm framework for mean-variance optimization in discounted Markov decision processes

This paper studies the risk-averse mean-variance optimization in infinite-horizon discounted Markov decision processes (MDPs). The involved variance metric concerns reward variability during whole process, and future deviations are to their present values. yields a function dependent on mean, this dependency renders traditional dynamic programming methods inapplicable since it suppresses crucial property—time-consistency. To deal with unorthodox problem, we introduce pseudo mean transform untreatable MDP standard one redefined form derive performance difference formula. With propose unified algorithm framework bilevel structure for optimization. unifies variety of algorithms several variance-related problems including, but not limited to, optimizations average MDPs. Furthermore, convergence analyses missing from literature can be complemented proposed as well. Taking value iteration an example, develop prove its local optimum aid Bellman local-optimality equation. Finally, conduct numerical experiment portfolio management validate algorithm.